Network accessible sites offer users a variety of experiences. For example, users may access such sites over a network such as, for example, websites accessed over the Internet, in order to view media, play games, purchase goods and/or services, access information, provide information, etc. Operators of such websites may wish to know the effect various changes to the website may have on user behavior. For example, operators of network accessible sites may wish to know what effect moving a graphical user interface (GUI) displayed on the site has on user behavior.
In order to predict what effect changes at a network accessible site may have on user behavior, operators may run experiments. For example, an experiment might be to move a GUI at a website. In order to determine the effect such a change may have, the experiment may be run as what is referred to as an A/B experiment where two groups are randomly established. A first group may be a control group, where the location of the GUI remains unchanged. A second group may be a treatment group, where the location of the GUI at the website is changed, for example, moved up on a page at the site. The operator of the site running the experiment may wish to know if the change in location of the GUI affects how much users interact with the GUI at the website.
Thus, the operator runs the experiment and at the end of the experiment compares the results from the two groups. However, in order to determine a length of time for running the experiment, historical data is generally used to determine the length of time, thereby delaying the start of the experiment. Additionally, in order to obtain enough data for there to be a statistically significant difference between the two groups, the length of time for running the experiment, based upon historical data, may be determined to be three months, six months or even a year, thereby requiring large amounts of processor cycles, computing resources, power.
Finally, in order for data collected by experiments such as those described above to be meaningful, “false negatives” with respect to usability of data obtained from experiments must be avoided, especially if the random assignment of users to the control group and the treatment group is handled as desired. Such false negatives cause an experiment to be needlessly run again, thereby delaying obtaining results of the experiment and requiring additional processor cycles, computing resources, power. These potential problems can create a burden for the operator of the website who may wish to know the effect of the potential change much sooner in order to implement the change in a timely manner and/or who may run an experiment that is believed to have unusable data, when the data is in fact usable.
The disclosure made herein is presented with respect to these and other considerations.